from sklearn_benchmarks.reporting.hp_match import HPMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HPMatchReporting(against_lib="onnx", config="config.yml")
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

WARNING! Mismatch between validation scores for 12 predictions (75.0%). See below.

	estimator	function	n_samples_train	n_samples	n_features	accuracy_score_sklearn	accuracy_score_onnx	diff_accuracy_scores
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.663	1.000	0.337
2	KNeighborsClassifier_brute_force	predict	100000	1	100	1.000	0.757	0.243
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.757	0.882	0.125
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.882	1.000	0.118
8	KNeighborsClassifier_brute_force	predict	100000	1	100	1.000	0.757	0.243
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.882	0.663	0.219
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.757	1.000	0.243
14	KNeighborsClassifier_brute_force	predict	100000	1	100	1.000	0.922	0.078
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	0.663	0.929	0.266
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	0.896	1.000	0.104
20	KNeighborsClassifier_brute_force	predict	100000	1	2	1.000	0.922	0.078
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	0.922	0.896	0.026

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.013	0.001	NaN	5.957	0.0	-1	1	NaN	16.762	0.113	0.663	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.013	0.001	NaN	6.199	0.0	-1	5	NaN	0.293	0.011	1.000	0.044	0.044	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.014	0.001	NaN	5.674	0.0	1	100	NaN	17.400	0.502	0.882	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	NaN	6.445	0.0	-1	100	NaN	0.294	0.008	1.000	0.042	0.042	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.001	NaN	6.560	0.0	1	5	NaN	4.965	0.014	0.896	0.002	0.002	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.013	0.001	NaN	6.386	0.0	1	1	NaN	0.225	0.008	1.000	0.056	0.056	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.000	NaN	0.431	0.0	-1	1	NaN	5.065	0.078	0.929	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.001	NaN	0.362	0.0	-1	5	NaN	0.225	0.006	1.000	0.020	0.020	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.124	0.217	NaN	0.000	0.002	-1	1	0.663	0.303	0.012	1.000	7.007	7.012	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.025	0.003	NaN	0.000	0.025	-1	1	1.000	16.605	0.059	0.757	0.002	0.002	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.700	0.059	NaN	0.000	0.003	-1	5	0.757	16.722	0.033	0.882	0.161	0.161	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.028	0.003	NaN	0.000	0.028	-1	5	1.000	0.307	0.011	1.000	0.092	0.092	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.111	0.038	NaN	0.000	0.002	1	100	0.882	0.299	0.015	1.000	7.069	7.078	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.001	NaN	0.000	0.023	1	100	1.000	16.489	0.104	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.666	0.107	NaN	0.000	0.003	-1	100	0.882	16.511	0.115	0.663	0.161	0.161	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.025	0.003	NaN	0.000	0.025	-1	100	1.000	0.295	0.006	1.000	0.084	0.084	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.067	0.015	NaN	0.000	0.002	1	5	0.757	0.222	0.006	1.000	9.305	9.308	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.000	NaN	0.000	0.023	1	5	1.000	5.068	0.149	0.922	0.005	0.005	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.385	0.012	NaN	0.001	0.001	1	1	0.663	5.007	0.026	0.929	0.277	0.277	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.001	NaN	0.000	0.022	1	1	1.000	0.223	0.006	1.000	0.100	0.100	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.602	0.031	NaN	0.000	0.002	-1	1	0.896	0.226	0.006	1.000	7.077	7.079	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.005	0.002	NaN	0.000	0.005	-1	1	1.000	4.994	0.073	0.922	0.001	0.001	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.305	0.029	NaN	0.000	0.002	-1	5	0.922	5.013	0.052	0.896	0.460	0.460	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.007	0.005	NaN	0.000	0.007	-1	5	1.000	0.224	0.006	1.000	0.031	0.031	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

WARNING! Mismatch between validation scores for 12 predictions (75.0%). See below.

	estimator	function	n_samples_train	n_samples	n_features	accuracy_score_sklearn	accuracy_score_onnx	diff_accuracy_scores
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.929	1.000	0.071
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	1.000	0.946	0.054
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.946	0.951	0.005
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.951	1.000	0.049
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	1.000	0.946	0.054
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.951	0.929	0.022
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.946	1.000	0.054
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	1.000	0.911	0.089
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.929	0.894	0.035
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.891	1.000	0.109
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	1.000	0.911	0.089
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.911	0.891	0.020

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	2.291	0.038	NaN	0.035	0.0	-1	1	NaN	138.071	0.000	0.929	0.017	0.017	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.027	0.042	NaN	0.026	0.0	-1	5	NaN	2.562	0.245	1.000	1.182	1.187	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.222	0.189	NaN	0.025	0.0	1	100	NaN	137.693	0.000	0.951	0.023	0.023	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.173	0.090	NaN	0.025	0.0	-1	100	NaN	2.528	0.248	1.000	1.255	1.261	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.116	0.071	NaN	0.026	0.0	1	5	NaN	0.055	0.014	0.891	56.513	58.415	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.137	0.056	NaN	0.026	0.0	1	1	NaN	0.005	0.000	1.000	588.923	590.818	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	NaN	0.015	0.0	-1	1	NaN	0.075	0.002	0.894	0.015	0.015	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	NaN	0.024	0.0	-1	5	NaN	0.005	0.001	1.000	0.124	0.124	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.841	1.153	NaN	0.000	0.001	-1	1	0.929	2.528	0.273	1.000	0.333	0.335	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.000	NaN	0.000	0.003	-1	1	1.000	137.375	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.944	0.346	NaN	0.000	0.001	-1	5	0.946	138.756	0.000	0.951	0.007	0.007	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.000	NaN	0.000	0.003	-1	5	1.000	2.505	0.220	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	4.863	0.483	NaN	0.000	0.005	1	100	0.951	2.523	0.223	1.000	1.927	1.935	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	NaN	0.000	0.003	1	100	1.000	136.950	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	2.913	0.312	NaN	0.000	0.003	-1	100	0.951	139.136	0.000	0.929	0.021	0.021	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.004	0.001	NaN	0.000	0.004	-1	100	1.000	2.565	0.218	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.538	0.458	NaN	0.000	0.002	1	5	0.946	0.005	0.000	1.000	300.396	301.331	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.001	NaN	0.000	0.001	1	5	1.000	0.055	0.005	0.911	0.026	0.026	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.855	0.388	NaN	0.000	0.001	1	1	0.929	0.070	0.003	0.894	12.126	12.138	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	NaN	0.000	0.001	1	1	1.000	0.005	0.000	1.000	0.172	0.173	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.029	0.016	NaN	0.001	0.000	-1	1	0.891	0.005	0.001	1.000	5.251	5.275	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.001	NaN	0.000	0.003	-1	1	1.000	0.054	0.002	0.911	0.050	0.050	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.025	0.001	NaN	0.001	0.000	-1	5	0.911	0.051	0.002	0.891	0.491	0.491	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	NaN	0.000	0.002	-1	5	1.000	0.005	0.001	1.000	0.397	0.398	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

WARNING! Mismatch between validation scores for 1 prediction (100.0%). See below.

	estimator	function	n_samples_train	n_samples	n_features	accuracy_score_sklearn	accuracy_score_onnx	diff_accuracy_scores
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.824	1.0	0.176

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	102.044	0.0	300	0.001	0.001	NaN	0.47	0.034	0.824	216.96	217.531	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.16	0.003	300	0.005	0.0	0.824	0.398	0.023	1.0	0.403	0.404	See	See